Driver condition estimating device, driver condition estimating method and computer program therefor

ABSTRACT

A driver condition estimating device includes circuitry configured to measure movement of the head of a driver output from a driver camera that photographs the driver and detect a sign of abnormality of the driver from the movement of the head. The circuitry determines existence of the sign of abnormality of the driver by calculating a periodic feature amount from time series data showing movement of the head of the driver, calculating coherence between the movement of the head of the driver and lateral acceleration acting on the head of the driver, calculating time series variation patterns from the obtained periodic feature amount and the obtained coherence, and comparing the obtained time series variation patterns with a predetermined threshold.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority to JP 2020-039035 filed on Mar.6, 2020, the entire contents of which are incorporated herein byreference.

TECHNICAL FIELD

The technique disclosed herein belongs to a field of technology forestimating condition of a driver who is driving a vehicle.

BACKGROUND ART

Recently, development of automated driving systems is promoted. Theapplicant of this application has a view that automated driving systemsare roughly classified into the following two types.

The first type is a system that an automobile autonomously carries apassenger to a destination without needing operation of a driver, whichis so-called “full driving automation” of an automobile. The second typeis an automated driving system based on a premise that a person drives,for example, in order to enjoy driving an automobile, while assuming thedriver is normally responsible for driving.

The second automated driving system is supposed to make an automobileautomatically substitute for a driver and perform automated driving,e.g., in the event that the driver falls ill, falls asleep, and thelike, and can no longer drive normally. For this purpose, earlydetection with high accuracy of occurrence of abnormality in a driver,in particular, occurrence of dysfunction or illness in a driver, is veryimportant from the point of view of improving a life-saving rate of adriver and securing safety of the driver and the surroundings.

Patent document 1 discloses a technique for detecting loss of theability to drive of a driver from abnormal direction of the face of thedriver as well as non-driving operation or abnormal driving operation.Patent Document 2 discloses a technique for determining occurrence ofabnormality in a driver when the head of the driver is moved greatly orslightly by an applied external force.

PRIOR ART DOCUMENTS Patent Documents

[Patent document 1] JP-B-6379720

[Patent document 2] JP-B-6361312

Non-Patent Documents

[Non-Patent document 1] T. Nakamura, et al., Multiscale Analysis ofIntensive Longitudinal Biomedical Signals and its Clinical Applications,Proceedings of the IEEE, Institute of Electrical and ElectronicsEngineers, 2016, vol. 104, pp. 242-261

[Non-Patent document 2] Mizuta et al., Fractal time series analysis ofpostural stability, Equilibrium Research, Japan Society for EquilibriumResearch, 2016, Vol. 75(3), pp. 154-161

SUMMARY OF THE DISCLOSURE

As disclosed in Patent documents 1 and 2, techniques for determiningloss of the ability to drive of a driver or determining abnormality in adriver on the basis of movement of the head of the driver, are alreadyknown. These techniques detect a condition in which the driver loses theability to drive after the driver falls ill. In order to more safelyconduct an action, such as emergency stop, at the time when abnormalityoccurs in a driver, it is preferable to detect the sign before thedriver loses the ability to drive. Early detection of the sign of losingthe ability to drive reduces a free running time of a vehicle due tooccurrence of an abnormality and makes it possible to more safelyconduct an action, such as emergency stop.

One or more embodiments disclosed herein enables early detection of asign of losing the ability to drive of a driver who is driving avehicle.

To solve the above problem, the technique disclosed herein includes adriver condition estimating device that is configured to estimatecondition of a driver who is driving a vehicle. The driver conditionestimating device includes a head movement measuring unit and adetector. The head movement measuring unit is configured to measuremovement of the head of the driver from output of a camera thatphotographs the driver. The detector is configured to detect a sign ofabnormality of the driver from the movement of the head measured by thehead movement measuring unit. The detector is further configured tocalculate a periodic feature amount from time series data showing themovement of the head of the driver and to calculate coherence betweenthe movement of the head of the driver and lateral acceleration actingon the head of the driver. The detector is further configured tocalculate time series variation patterns from the calculated periodicfeature amount and the calculated coherence and to compare thecalculated time series variation patterns with a predetermined thresholdto determine existence of the sign of abnormality of the driver.

This technique focuses on a homeostatic maintaining function of a humanbody and detects a sign of abnormality from movement of the head of adriver. Homeostasis is a function to maintain conditions fromdisturbances. As for movement of the head, homeostasis makes a driver tomaintain the posture of the head during driving. The head of a driverirregularly moves due to the maintaining function of homeostasis in anormal condition but moves slightly and stably in an illness occurringcondition. Moreover, the head moves periodically in a condition, calleda “critical slowing down” condition, when the condition is changed fromthe normal condition to the illness occurring condition. In view ofthis, the inventors of this application came to a thought that aperiodic feature amount of movement of the head can be used to detect asign of abnormality of a driver. Moreover, results of furtherexperiments and investigations showed that the periodicity of movementof the head frequently appears even in the normal condition. However, byclassifying time series variation patterns of periodic feature amountsinto patterns is effective for distinguishing the normal condition froman abnormality sign appearing condition. In addition, results of furtherinvestigation revealed that the periodicity of movement of the headtends to appear in a case in which lateral acceleration greatly acts onthe head of a driver, such as traveling around a corner.

Thus, this technique detects a sign of abnormality of a driver frommovement of the head of the driver. The head movement measuring unitmeasures movement of the head of a driver from output of the camera thatphotographs the driver. The detector for detecting the sign ofabnormality of a driver determines existence of the sign of abnormalityof the driver by the following processes: calculation of a periodicfeature amount from time series data showing movement of the head of thedriver, calculation of coherence between the movement of the head of thedriver and lateral acceleration acting on the head of the driver,calculation of time series variation patterns from the calculatedperiodic feature amount and the calculated coherence, and comparison ofthe obtained time series variation patterns with a predeterminedthreshold. This enables determination in consideration of thecorrelation between the movement of the head and the lateralacceleration as an external factor, whereby the sign of abnormality ofthe driver can be detected earlier with high accuracy. In addition, itis possible to measure movement of the head of the driver from aphotographic image of the camera placed in the vehicle interior.Therefore, this technique enables early detection of a sign of illnessthat can cause loss of the ability to drive, by using an existingonboard sensor.

The detector may be configured to determine whether the lateralacceleration acting on the head of the driver exceeds a predeterminedvalue, from output of a sensor for measuring a movement state of thevehicle.

Thus, the detector easily recognizes the scene where the sign ofabnormality is difficult to detect.

The detector may be configured to dimensionally reduce time series dataof the calculated periodic feature amount and of the calculatedcoherence by using a nonlinear dimensionality reduction method to obtaintwo-dimensional data as the time series variation patterns. The detectormay be further configured to determine existence of the sign ofabnormality of the driver from the obtained two-dimensional data byusing a determination line in a two-dimensional map as the predeterminedthreshold.

This enables easy determination of existence of the sign of abnormalityof the driver from the time series data of both the periodic featureamount of the movement of the head and the coherence.

The technique disclosed herein also includes a driver conditionestimating method for estimating condition of a driver who is driving avehicle. The driver condition estimating method includes calculating aperiodic feature amount from time series data showing movement of thehead of the driver, and calculating coherence between the movement ofthe head of the driver and lateral acceleration acting on the head ofthe driver. The driver condition estimating method also includescalculating time series variation patterns from the calculated periodicfeature amount and the calculated coherence, and comparing thecalculated time series variation patterns with a predetermined thresholdto determine existence of a sign of abnormality of the driver.

This technique determines existence of the sign of abnormality of thedriver by these processes: calculation of a periodic feature amount fromtime series data showing movement of the head of the driver, calculationof coherence between the movement of the head of the driver and lateralacceleration acting on the head of the driver, calculation of timeseries variation patterns from the calculated periodic feature amountand the calculated coherence, and comparison of the obtained time seriesvariation patterns with a predetermined threshold. This enablesdetermination in consideration of the correlation between the movementof the head and the lateral acceleration as an external factor, wherebythe sign of abnormality of the driver can be detected earlier with highaccuracy.

As described above, the technique disclosed herein enables earlydetection of a sign of losing the ability to drive of a driver who isdriving a vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a conceptual diagram showing an overview of a techniqueaccording to this disclosure.

FIG. 2 shows relationships between homeostasis of movement of a head andillness. In FIG. 2, (a) shows a normal condition, (b) shows a criticalslowing down condition, and (c) is an illness occurring condition.

FIG. 3 shows examples of time series data of a pitch angle and a rollangle of a head.

FIG. 4 is a graph showing distributions of autocorrelation indexes ofthe pitch angle and the roll angle of the head.

FIG. 5 is a graph showing time series variations of the autocorrelationindexes.

FIG. 6 is a graph showing result of classifying time series variationpatterns of the autocorrelation indexes.

FIGS. 7A to 7C show time series data of a pitch angle of a head. FIG. 7Ais time series data obtained in the normal condition during straighttraveling, FIG. 7B is time series data obtained in an abnormalitysimulating condition, and FIG. 7C is time series data obtained in thenormal condition during traveling a corner.

FIGS. 8A and 8B are data of traveling experiments. FIG. 8A is dataobtained in entering a corner from a straight section, and FIG. 8B isdata obtained while the condition is changed from the normal conditionto the abnormality simulating condition.

FIG. 9 is a graph showing a time series variation of coherence betweenmovement of a head and lateral acceleration.

FIG. 10 is a graph showing result of classifying time series variationpatterns of coherence and autocorrelation indexes.

FIG. 11 is an example of a configuration of an onboard system includinga driver condition estimating device according to an embodiment.

FIG. 12 is an example of a flow of processing of estimating condition ofa driver in the embodiment.

FIG. 13 is a block diagram of computer-based circuitry that may be usedto implement control features of the present disclosure.

DETAILED DESCRIPTION

Hereinafter, an exemplary embodiment will be detailed with reference tothe drawings.

FIG. 1 is a conceptual diagram showing an overview of a techniqueaccording to this disclosure. Patterns of condition change that causes adriver to lose the ability to drive are summarized into three cases. Acase “A” is a pattern in which one of functions of perception, judgment,and movement deteriorates first, a case “B” is a pattern in whichmultiple functions gradually deteriorate, and a case “C” is a pattern inwhich consciousness is suddenly lost. Among them, in the cases “A” and“B”, the level of the ability to drive of a driver is gradually loweredto cause a condition in which the ability to drive is lost, afterillness occurs, as shown in FIG. 1. Thus, detecting the condition inwhich the ability to drive is deteriorated makes it possible to detect asign of losing the ability to drive of the driver. In the condition inwhich the sign of losing the ability to drive is detected, an emergencyresponse can be subsequently conducted, for example, such that anintention of the driver is confirmed and the vehicle is evacuated to ashoulder of a road by automatic traveling control.

The technique according to this disclosure focuses on a homeostaticmaintaining function of a human body and detects a sign of losing theability to drive from movement of the head of a driver.

FIG. 2 shows relationships between homeostasis of movement of a head andillness (refer to Non-Patent Documents 1 and 2). Each of the graphs inFIG. 2 shows movement of a head as viewed from above. In FIG. 2, (a)shows a normal condition in which homeostasis is maintained, (b) shows acondition between the normal condition and an illness occurringcondition, which is called a “critical slowing down” condition, and (c)shows the illness occurring condition. Note that the graphs (a) and (c)in FIG. 2 are quoted from Non-Patent Document 1.

Human bodies have a function, called “homeostasis”, that maintainsconditions constant against disturbances. Homeostasis of movement of ahead is a characteristic of maintaining the posture of the head duringdriving. As shown in (a) of FIG. 2, in the normal condition, the headirregularly moves due to the maintaining function of homeostasis. On theother hand, as shown in (c) of FIG. 2, when illness occurs, the headmoves slightly, and the movement is stable. Meanwhile, as shown in (b)of FIG. 2, in the critical slowing down condition in which the conditionis changed from the normal condition to the illness occurring condition,the head is presumed to move in a periodic manner or in anautocorrelative manner (refer to Non-Patent Document 2).

The inventors of this application focused on the above-described findingand realized that a sign of abnormality can be detected by detecting acondition change to the critical slowing down condition, from themovement of the head of a driver. More specifically, the inventorsconcluded that a periodic feature amount of the movement of a head canbe used to detect a sign of abnormality of a driver.

The inventors of this application performed experiments as follows. Adriver was made to drive a test course, and movement of the head of thedriver was measured during driving. The driver was made to drivenormally as usual (this corresponds to the normal condition), and then,a signal and an abnormality simulation task or complicated mentalarithmetic were given to the driver (this corresponds to the abnormalitysign appearing condition). A pitch angle that is an angle in afront-back direction of the head and a roll angle that is an angle in aright-left direction of the head, were measured from an image of acamera that photographs the driver, as data representing movement of thehead. Thereafter, time series data of the pitch angle and the roll angleof the head were subjected to detrended fluctuation analysis (DFA),whereby autocorrelation indexes or scaling exponents a were calculated.DFA is a method for investigating scaling by removing a component thatvery slowly changes, which is so-called “trend”. The autocorrelationindex or the scaling exponent a is an example of a feature amountshowing periodicity of data.

FIG. 3 shows examples of the time series data of the pitch angle and theroll angle of a head. FIG. 4 is a graph showing distributions of theautocorrelation indexes of the pitch angle and the roll angle of thehead. The lateral axis represents the autocorrelation index of the rollangle, whereas the longitudinal axis represents the autocorrelationindex of the pitch angle. A smaller autocorrelation index value showsstronger autocorrelation, and a greater autocorrelation index valueshows weaker autocorrelation.

FIG. 4 shows that, in the abnormality sign appearing condition, theautocorrelation indexes are distributed toward the values showingstronger autocorrelation indexes, compared with those in the normalcondition. These experimental results agree with the above-describedfinding that a head moves irregularly in the normal condition but movesperiodically in the critical slowing down condition. However, as may beseen from FIG. 4 that the distribution in the normal condition spreadsover a relatively large range and largely overlaps the distribution inthe abnormality sign appearing condition. Thus, it is not necessarilyeasy to determine the abnormality sign appearing condition by using onlythe autocorrelation index.

For this reason, the inventors of this application focused on a timeseries variation pattern of an autocorrelation index. As shown in FIG.5, time series data of autocorrelation indexes were aligned in timesequence, and these data were classified into time series variationpatterns. Herein, the method of the pattern classification used anonlinear dimensionality reduction method, and more specifically,uniform manifold approximation and projection (UMAP).

FIG. 6 shows result of classifying the time series variation patterns ofthe autocorrelation indexes. In FIG. 6, the time series variationpatterns are dimensionally reduced to two dimensions by UMAP and aremapped into a two-dimensional map. In FIG. 6, the distribution in thenormal condition and the distribution in the abnormality sign appearingcondition are separated from each other more than in FIG. 4. Adetermination line, LTH, shown in FIG. 6, was obtained by using asupport vector machine on the two-dimensional data. The determinationline LTH may include a plurality of line segments to divide thetwo-dimensional map into regions in which a region above thedetermination line LTH is classified as normal while a region below thedetermination line LTH is classified as abnormal. Use of thisdetermination line LTH achieved a wrong determination rate of 15% indetermining the abnormality sign appearing condition. In particular, forthis example determination line LTH, more normal conditions areincorrectly classified as abnormal conditions than abnormal conditionsare classified as normal conditions. The exact determination line LTHmay be adjusted according to the acceptability of these errors, e.g.,may minimize incorrect classification of abnormal conditions.

Accordingly, the sign of abnormality of a driver can be detected by thefollowing processes: calculation of a periodic feature amount from timeseries data showing movement of the head of the driver, calculation of atime series variation pattern from the calculated periodic featureamount, and comparison of the calculated time series variation patternwith a predetermined threshold. As may be seen in FIG. 6, thispredetermined threshold is not a single value, but is a predetermineddetermination line LTH such that the determination of the conditiondepends on which side of the predetermined determination line LTH thedetected conditions are on.

Moreover, the inventors of this application investigated how to furtherincrease accuracy of determining the sign of abnormality. The result offurther investigation revealed that there is a case in which movement ofthe head is similar to that in the abnormality sign appearing conditionalthough a driver is in the normal condition.

FIGS. 7A to 7C are time series data of a pitch angle of a head obtainedin experiments performed by the inventors of this application. FIG. 7Ashows a case in which a driver drives straight in the normal condition,FIG. 7B is a case in which the driver drives in the abnormalitysimulating condition, and FIG. 7C is a case in which the driver drives acorner in the normal condition. Each graph shows raw data and a movingaverage of the raw data. FIG. 7A shows that the head moves irregularlyin the case in which the driver drives straight in the normal condition,and FIG. 7B shows that the head moves regularly in the case in which thedriver drives in the abnormality simulating condition. On the otherhand, FIG. 7C shows that the head moves regularly in the case in whichthe driver drives a corner in the normal condition.

That is, in the case in FIG. 7C, although the driver is in the normalcondition, the autocorrelation index of the head pitch angle data islarge, and therefore, it is difficult to distinguish from the case inFIG. 7B. It is presumed that the cause of the regular movement of thehead of the driver during traveling a corner is lateral accelerationacting on the head.

FIGS. 8A and 8B are data obtained by traveling experiments. FIG. 8A isdata obtained when a vehicle enters a corner from a straight section ofa circuit track, and FIG. 8B is data obtained when the condition of adriver is changed from the normal condition to the abnormalitysimulating condition. In FIGS. 8A and 8B, upper graphs show time seriesvariations of vehicle lateral acceleration and a head roll angle, andlower graphs show a time series variation of coherence or of across-correlation index between the vehicle lateral acceleration and thehead roll angle. The coherence is obtained by correlating a powerspectrum that is obtained from the time series variation of the vehiclelateral acceleration with a power spectrum that is obtained from thetime series variation of the head roll angle. In the graph of thecoherence, the lateral axis represents time, whereas the longitudinalaxis represents frequency, and thinner color represents highercorrelation, whereas thicker color represents lower correlation.

The upper graph in FIG. 8A shows that the head moves in accordance withchange in the lateral acceleration after the vehicle enters the corner.That is, the driver corrects the head in a direction opposite to thedirection of the lateral acceleration when the head starts to be swayedby the applied lateral acceleration. Thus, correlation between thevehicle lateral acceleration and the head roll angle gradually appearsafter the vehicle enters the corner. On the other hand, in the case inFIG. 8B, the head roll angle is not linked with change in the vehiclelateral acceleration, and coherence is not greatly changed.

These experimental results revealed that periodicity of movement of ahead tends to appear when lateral acceleration greatly acts on the headof a driver, such as going around a corner. The correlation appearsmainly at a frequency of 2 Hz or lower. This reflects a frequencyresponse of the head to the external force.

In view of this, the inventors of this application realized that, inorder to further increase determination accuracy of the sign ofabnormality, not only the autocorrelation index of movement of a headmay be calculated, but also an index of cross-correlation relative to anexternal factor, which is lateral acceleration herein, as featureamounts, and to classify the feature amounts into patterns. That is, asshown in FIG. 9, time series data of coherence was aligned in timesequence, and this aligned data was combined with the time series dataof the autocorrelation index. Then, this combined data was classifiedinto patterns. Herein, data at 1 Hz frequency, at which correlation withrespect to the vehicle lateral acceleration tends to appear, and data at4 Hz frequency, at which correlation with respect to the vehicle lateralacceleration does not tend to appear, were used as the time series dataof coherence. The method of the pattern classification used a nonlineardimensionality reduction method, and more specifically, UMAP.

FIG. 10 shows result of classifying time series variation patterns ofcoherence and autocorrelation indexes. In FIG. 10, as in the case inFIG. 6, the time series variation patterns are dimensionally reduced totwo dimensions by UMAP and are mapped into a two-dimensional map. Adetermination line, LTH2, shown in FIG. 10, was obtained by using asupport vector machine on the two-dimensional data. The determinationline LTH2 may include a plurality of segments to divide thetwo-dimensional map into regions in which a region above thedetermination line LTH2 is classified as normal while a region below thedetermination line LTH2 is classified as abnormal. Use of thisdetermination line LTH2 achieved a wrong determination rate of 1% indetermining the abnormality sign appearing condition. Thus, as comparedto using the determination rate LTH and the two-dimensional map of FIG.6, the wrong determination rate may be reduced. In particular, for thisexample determination line LTH2, more normal conditions are incorrectlyclassified as abnormal conditions than abnormal conditions areclassified as normal conditions. The exact determination line LTH2 maybe adjusted according to the acceptability of these errors, e.g., mayminimize incorrect classification of abnormal conditions.

Accordingly, the sign of abnormality of a driver can be furtheraccurately detected by the following processes: calculation of aperiodic feature amount from time series data showing movement of thehead of the driver, calculation of coherence between the movement of thehead of the driver and lateral acceleration acting on the head of thedriver, calculation of time series variation patterns from thecalculated periodic feature amount and the calculated coherence, andcomparison of the calculated time series variation patterns with apredetermined threshold to perform determination in consideration of thecorrelation between the movement of the head and the lateralacceleration as an external factor. As may be seen in FIG. 10, thispredetermined threshold is not a single value, but is a predetermineddetermination line LTH2 such that the determination of the conditiondepends on which side of the predetermined determination line LTH2 thedetected conditions are on.

The following describes a driver condition estimating device accordingto this embodiment.

FIG. 11 is a block diagram showing an example of a configuration of anonboard system including the driver condition estimating deviceaccording to this embodiment. In the onboard system in FIG. 11, a drivercamera 10, a speaker 11, an information presenting unit 12, a switch 13,and a microphone 14 are mounted in the vehicle interior. An informationprocessor 20 is composed of, for example, a single IC chip having aprocessor and a memory or multiple IC chips having a processor and amemory. A vehicle stop controller 40 controls to automatically evacuateand stop the vehicle at a shoulder of a road, upon receiving aninstruction from the information processor 20. A vehicle accelerationsensor 50 is provided to the vehicle.

Optionally, the information processor 20 may include a processor 835 andother circuitry in system 800 of FIG. 13, which may be implemented as asingle processor-based system, or a distributed processor based system,including remote processing, such as cloud based processing.

The driver camera 10 is placed, for example, at an inner side of awindshield and photographs the state of the vehicle interior, includinga driver. The photographic image obtained by the driver camera 10 istransmitted to the information processor 20 by, for example, an onboardnetwork.

In the information processor 20, a head movement measuring unit 21measures movement of the head of the driver from the photographic imageobtained by the driver camera 10. For example, the head of the driver isrecognized in the image, and tilt angles of the head, e.g., a pitchangle and a roll angle, are measured. The processing in the headmovement measuring unit 21 can be implemented by an existing imageprocessing technique. The processing performed by the head movementmeasuring unit 21 provides time series data of movement of the head, asshown in FIG. 3. The time series data of movement of the head istransmitted to a detector 30 for detecting the sign of abnormality of adriver.

The detector 30 includes a periodic feature amount arithmetic unit 31, atime series variation pattern arithmetic unit 32, an abnormalitydetermining unit 33, an abnormality determination threshold database 34,and a coherence arithmetic unit 35. The periodic feature amountarithmetic unit 31 calculates periodic feature amounts from the timeseries data of movement of the head obtained by the head movementmeasuring unit 21. In a specific example, autocorrelation indexes orscaling exponents a are calculated as the periodic feature amounts bydetrended fluctuation analysis (DFA). The periodic feature amountarithmetic unit 31 provides time series data of the periodic featureamounts, as shown in FIG. 5.

The coherence arithmetic unit 35 calculates coherence between themovement of the head of the driver and lateral acceleration acting onthe head of the driver by using output of the vehicle accelerationsensor 50 and the time series data of the movement of the head, which isobtained by the head movement measuring unit 21. For example,acceleration in the right-left direction of the vehicle, which isrecognized from the output of the vehicle acceleration sensor 50, isused as lateral acceleration acting on the head of the driver, and apower spectrum is calculated from the time series data of this lateralacceleration. Moreover, a power spectrum is calculated from the timeseries data of the movement of the head, and coherence is obtained bycorrelating these power spectra. Thereafter, time series data at afrequency at which correlation with respect to the lateral accelerationtends to appear, for example, at 1 Hz, and time series data at afrequency at which correlation with respect to the lateral accelerationdoes not tend to appear, for example, at 4 Hz, are obtained.

The time series variation pattern arithmetic unit 32 aligns the timeseries data of the periodic feature amounts, which is obtained by theperiodic feature amount arithmetic unit 31, and the time series data ofthe coherence, which is obtained by the coherence arithmetic unit 35, intime sequence. The time series variation pattern arithmetic unit thencalculates time series variation patterns from combination of thealigned time series data and classifies them into patterns. In aspecific example, the time series data of combination of the periodicfeature amounts and the coherence are dimensionally reduced to beconverted into two-dimensional data by using UMAP, which is one ofnonlinear dimensionality reduction methods.

The abnormality determining unit 33 compares the data obtained by thetime series variation pattern arithmetic unit 32, with a thresholdstored in the abnormality determination threshold database 34, todetermine whether the sign of abnormality appears in the driver. Forexample, the determination line LTH2 of the two-dimensional map in FIG.10 corresponds to the threshold stored in the abnormality determinationthreshold database 34. The abnormality determining unit 33 determineswhether the sign of abnormality appears in the driver, by examining onwhich side of the determination line LTH2 the data obtained by the timeseries variation pattern arithmetic unit 32 exists in thetwo-dimensional map.

The detector 30 outputs a request for inquiry to the driver, to aninquiring unit 22 in response to the abnormality determining unit 33determining that the sign of abnormality appears.

The inquiring unit 22 inquires the driver upon receiving the request forinquiry to the driver from the detector 30. This inquiry is made inorder to confirm the intention of the driver to emergently evacuate thevehicle by automated driving. The inquiry is performed, for example, byusing voice sound from the speaker 11 or by showing it via theinformation presenting unit 12, such as a monitor.

A response detector 23 detects a response of the driver to the inquiryprovided by the inquiring unit 22. The driver responds, for example, byoperating the switch 13 or by speaking via the microphone 14. When theintention of the driver is confirmed or when no response is receivedfrom the driver, the information processor 20 instructs the vehicle stopcontroller 40 to automatically evacuate and stop the vehicle at ashoulder of a road.

The driver condition estimating device according to this embodimentincludes at least the head movement measuring unit 21 and the detector30 of the information processor 20. In some cases, the driver conditionestimating device according to this disclosure also includes the drivercamera 10.

FIG. 12 is a flowchart showing an example of a flow of processing ofestimating condition of a driver. First, the head movement measuringunit 21 recognizes an image of the head of a driver in a photographicimage obtained by the driver camera 10 and calculates tilt angles, whichare a pitch angle and a roll angle herein, of the recognized head (S11).This calculation is performed at every 100 ms, for example. After tiltangle data of the head is accumulated at a specified amount or more (YESin S12), the periodic feature amount arithmetic unit 31 of the detector30 calculates periodic feature amounts from time series data of the tiltangles of the head (S13). In a specific example, autocorrelation indexesor scaling exponents a are calculated as the periodic feature amounts byDFA. The periodic feature amounts are calculated by changing a timerange of the target tilt angle data. For example, the periodic featureamounts are calculated by using 256 pieces of tilt angle data at every100 ms.

Moreover, the coherence arithmetic unit 35 of the detector 30 calculatescoherence between the movement of the head of the driver and lateralacceleration acting on the head of the driver by using output of thevehicle acceleration sensor 50 and the time series data of the tiltangles of the head (S14).

After data of the periodic feature amounts and of the coherence areaccumulated at a specified amount or more (YES in S15), the time seriesvariation pattern arithmetic unit of the detector 30 calculates timeseries variation patterns from combined data of the periodic featureamounts and the coherence (S16). In a specific example, the time seriesdata of the periodic feature amounts and of the coherence aredimensionally reduced to be converted into two-dimensional data by usingUMAP, which is one of nonlinear dimensionality reduction methods. Thetime series variation patterns are calculated by changing time ranges ofthe target periodic feature amount data and the target coherence data.For example, the time series variation patterns are calculated by using256 pieces of periodic feature amount data and of coherence data atevery 100 ms.

Thereafter, the abnormality determining unit 33 of the detector 30acquires a threshold held in the abnormality determination thresholddatabase 34 (S17). In a specific example, information of a determinationline in a two-dimensional map of classified patterns, such as thedetermination line LTH2 shown in FIG. 10, is acquired as the threshold.Then, the two-dimensional data of the time series variation patternsobtained in step S16 and the determination line are compared with eachother (S18). The abnormality determining unit 33 determines that thesign of abnormality appears in the case in which the two-dimensionaldata exists on an abnormality sign side beyond the determination line(YES in S18). At this time, the detector 30 outputs a request forinquiry to the driver, to the inquiring unit 22 (S19).

Alternatively, for example, determination of the sign of abnormality maybe performed such that the sign of abnormality is determined asappearing in a case in which multiple time series variation patternsexist on the abnormality sign side beyond the determination line in aconsecutive manner. In another example, the sign of abnormality may bedetermined as appearing in a case in which time series variationpatterns exist on the abnormality sign side beyond the determinationline, at an amount exceeding a predetermined ratio in a predeterminedtime period.

Thus, in this embodiment, the head movement measuring unit 21 measuresmovement of the head of a driver from output of the driver camera 10that photographs the driver. The detector 30 for detecting the sign ofabnormality of a driver determines existence of the sign of abnormalityof the driver by these processes: calculation of a periodic featureamount from time series data showing movement of the head of the driver,calculation of coherence between the movement of the head of the driverand lateral acceleration acting on the head of the driver, calculationof time series variation patterns from the periodic feature amount andthe coherence, and comparison of the obtained time series variationpatterns with a predetermined threshold. This enables determination inconsideration of the correlation between the movement of the head andthe lateral acceleration as an external factor, whereby the sign ofabnormality of the driver can be detected earlier with high accuracy. Inaddition, it is possible to measure movement of the head of the driverfrom a photographic image of the driver camera 10 placed in the vehicleinterior, and therefore, this technique enables early detection of asign of illness that can cause loss of the ability to drive, by using anexisting onboard sensor.

The threshold that is stored in the abnormality determination thresholddatabase 34 can be customized in accordance with a driver. For example,the threshold is initially set to a standard value, but is modifiedbased on accumulated time series variation patterns in the normalcondition, among time series variation patterns of periodic featureamounts and of coherence that are calculated and accumulated by thedetector 30 during traveling of the vehicle. For example, thedetermination line in the two-dimensional map may be modified inaccordance with distribution of time series variation patterns when adriver is in the normal condition.

The coherence arithmetic unit 35 calculates time series data atfrequencies of 1 Hz and 4 Hz as the time series data of the coherence inthe foregoing embodiment, but the frequency of the time series data tobe calculated is not limited to them. The 1 Hz frequency is an exampleof the frequency at which correlation between movement of a head andlateral acceleration tends to appear. The 4 Hz frequency is an exampleof the frequency at which correlation between movement of a head andlateral acceleration does not tend to appear. In one example, thecoherence arithmetic unit 35 may calculate time series data at three ormore kinds of frequencies or may calculate time series data at one kindof frequency. Still, the time series data of the coherence that iscalculated by the coherence arithmetic unit 35 preferably includes timeseries data at a frequency of 2 Hz or lower, at which correlationbetween movement of a head and lateral acceleration tends to appear.

In the foregoing embodiment, autocorrelation indexes or scalingexponents a are calculated as periodic feature amounts, from time seriesdata of movement of the head, by DFA. However, the method forcalculating a periodic feature amount is not limited to this. Forexample, fast Fourier transform (FFT) may be used, or simpleautocorrelation may be calculated.

The time series data of the periodic feature amounts and of thecoherence are dimensionally reduced to be converted into two-dimensionaldata by using UMAP in the foregoing embodiment, but the calculation ofthe time series variation patterns is not limited to this. For example,the time series data of the periodic feature amounts and of thecoherence may be dimensionally reduced to be converted intothree-dimensional data by using UMAP. In this case, for example, atwo-dimensional plane that is obtained by a support vector machine isused as the threshold for determining existence of the sign ofabnormality. Alternatively, the time series data of the periodic featureamounts and of the coherence may be dimensionally reduced by a methodother than UMAP. For example, a manifold learning method other thanUMAP, such as locally linear embedding (LLE) or t-distributed stochasticneighbor embedding (t-SNE), may also be used. Moreover, a generaldimensionality reduction method, e.g., principal component analysis(PCA), can also be employed.

Embodiments disclosed herein are not limited to automobiles. Forexample, embodiments disclosed herein may be used to detect the sign ofabnormality of a driver of a vehicle other than an automobile, such as atrain.

The technique according to this disclosure may be implemented by anembodiment other than a single information processor, in some cases. Forexample, the detector 30 may be implemented by an information processorseparately from the head movement measuring unit 21. In another example,the function of the detector 30 may be implemented by an informationprocessor that is not mounted on a vehicle, such as a smart phone or atablet carried by a driver. Alternatively, one or more or all ofarithmetic operations that are performed by the detector 30, may beexecuted by cloud computing.

The following description relates to a computer environment in whichembodiments of the present disclosure may be implemented. Thisenvironment may include an embedded computer environment, localmulti-processor embodiment, remote (e.g., cloud-based) environment, or amixture of all the environments.

FIG. 13 illustrates a block diagram of a computer that may implement thevarious embodiments described herein. The present disclosure may beembodied as a system, a method, and/or a computer program product. Thecomputer program product may include a computer readable storage mediumon which computer readable program instructions are recorded that maycause one or more processors to carry out aspects of the embodiment.

The non-transitory computer readable storage medium may be a tangibledevice that can store instructions for use by an instruction executiondevice (processor). The computer readable storage medium may be, forexample, but is not limited to, an electronic storage device, a magneticstorage device, an optical storage device, an electromagnetic storagedevice, a semiconductor storage device, or any appropriate combinationof these devices. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes each of the following (andappropriate combinations): flexible disk, hard disk, solid-state drive(SSD), random access memory (RAM), read-only memory (ROM), erasableprogrammable read-only memory (EPROM or Flash), static random accessmemory (SRAM), compact disc (CD or CD-ROM), digital versatile disk (DVD)and memory card or stick. A computer readable storage medium, as used inthis disclosure, is not to be construed as being transitory signals perse, such as radio waves or other freely propagating electromagneticwaves, electromagnetic waves propagating through a waveguide or othertransmission media (e.g., light pulses passing through a fiber-opticcable), or electrical signals transmitted through a wire.

Computer readable program instructions described in this disclosure canbe downloaded to an appropriate computing or processing device from acomputer readable storage medium or to an external computer or externalstorage device via a global network (i.e., the Internet), a local areanetwork, a wide area network and/or a wireless network. The network mayinclude copper transmission wires, optical communication fibers,wireless transmission, routers, firewalls, switches, gateway computersand/or edge servers. A network adapter card or network interface in eachcomputing or processing device may receive computer readable programinstructions from the network and forward the computer readable programinstructions for storage in a computer readable storage medium withinthe computing or processing device.

Computer readable program instructions for carrying out operations ofthe present disclosure may include machine language instructions and/ormicrocode, which may be compiled or interpreted from source code writtenin any combination of one or more programming languages, includingassembly language, Basic, Fortran, Java, Python, R, C, C++, C# orsimilar programming languages. The computer readable programinstructions may execute entirely on a user's personal computer,notebook computer, tablet, or smartphone, entirely on a remote computeror compute server, or any combination of these computing devices. Theremote computer or compute server may be connected to the user's deviceor devices through a computer network, including a local area network ora wide area network, or a global network (i.e., the Internet). In someembodiments, electronic circuitry including, for example, programmablelogic circuitry, field-programmable gate arrays (FPGA), or programmablelogic arrays (PLA) may execute the computer readable programinstructions by using information from the computer readable programinstructions to configure or customize the electronic circuitry, inorder to perform aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference toflow diagrams and block diagrams of methods, apparatus (systems), andcomputer program products according to embodiments of the disclosure. Itwill be understood by those skilled in the art that each block of theflow diagrams and block diagrams, and combinations of blocks in the flowdiagrams and block diagrams, can be implemented by computer readableprogram instructions.

The computer readable program instructions that may implement thesystems and methods described in this disclosure may be provided to oneor more processors (and/or one or more cores within a processor) of ageneral purpose computer, special purpose computer, or otherprogrammable apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmableapparatus, create a system for implementing the functions specified inthe flow diagrams and block diagrams in the present disclosure. Thesecomputer readable program instructions may also be stored in a computerreadable storage medium that can direct a computer, a programmableapparatus, and/or other devices to function in a particular manner, suchthat the computer readable storage medium having stored instructions isan article of manufacture including instructions which implement aspectsof the functions specified in the flow diagrams and block diagrams inthe present disclosure.

The computer readable program instructions may also be loaded onto acomputer, other programmable apparatus, or other device to cause aseries of operational steps to be performed on the computer, otherprogrammable apparatus or other device to produce a computer implementedprocess, such that the instructions which execute on the computer, otherprogrammable apparatus, or other device implement the functionsspecified in the flow diagrams and block diagrams in the presentdisclosure.

FIG. 13 is a functional block diagram illustrating a networked system800 of one or more networked computers and servers. In an embodiment,the hardware and software environment illustrated in FIG. 13 may providean exemplary platform for implementation of the software and/or methodsaccording to the present disclosure.

Referring to FIG. 13, a networked system 800 may include, but is notlimited to, computer 805, network 810, remote computer 815, web server820, cloud storage server 825 and computer server 830. In someembodiments, multiple instances of one or more of the functional blocksillustrated in FIG. 13 may be employed.

Additional detail of computer 805 is shown in FIG. 13. The functionalblocks illustrated within computer 805 are provided only to establishexemplary functionality and are not intended to be exhaustive. And whiledetails are not provided for remote computer 815, web server 820, cloudstorage server 825 and compute server 830, these other computers anddevices may include similar functionality to that shown for computer805.

Computer 805 may be built into the automobile, a personal computer (PC),a desktop computer, laptop computer, tablet computer, netbook computer,a personal digital assistant (PDA), a smart phone, or any otherprogrammable electronic device capable of communicating with otherdevices on network 810.

Computer 805 may include processor 835, bus 837, memory 840,non-volatile storage 845, network interface 850, peripheral interface855 and display interface 865. Each of these functions may beimplemented, in some embodiments, as individual electronic subsystems(integrated circuit chip or combination of chips and associateddevices), or, in other embodiments, some combination of functions may beimplemented on a single chip (sometimes called a system on chip or SoC).

Processor 835 may be one or more single or multi-chip microprocessors,such as those designed and/or manufactured by Intel Corporation,Advanced Micro Devices, Inc. (AMD), Arm Holdings (Arm), Apple Computer,etc. Examples of microprocessors include Celeron, Pentium, Core i3, Corei5 and Core i7 from Intel Corporation; Opteron, Phenom, Athlon, Turionand Ryzen from AMD; and Cortex-A, Cortex-R and Cortex-M from Arm.

Bus 837 may be a proprietary or industry standard high-speed parallel orserial peripheral interconnect bus, such as ISA, PCI, PCI Express(PCI-e), AGP, and the like.

Memory 840 and non-volatile storage 845 may be computer-readable storagemedia. Memory 840 may include any suitable volatile storage devices suchas Dynamic Random Access Memory (DRAM) and Static Random Access Memory(SRAM). Non-volatile storage 845 may include one or more of thefollowing: flexible disk, hard disk, solid-state drive (SSD), read-onlymemory (ROM), erasable programmable read-only memory (EPROM or Flash),compact disc (CD or CD-ROM), digital versatile disk (DVD) and memorycard or stick.

Program 848 may be a collection of machine readable instructions and/ordata that is stored in non-volatile storage 845 and is used to create,manage and control certain software functions that are discussed indetail elsewhere in the present disclosure and illustrated in thedrawings. In some embodiments, memory 840 may be considerably fasterthan non-volatile storage 845. In such embodiments, program 848 may betransferred from non-volatile storage 845 to memory 840 prior toexecution by processor 835.

Computer 805 may be capable of communicating and interacting with othercomputers via network 810 through network interface 850. Network 810 maybe, for example, a local area network (LAN), a wide area network (WAN)such as the Internet, or a combination of the two, and may includewired, wireless, or fiber optic connections. In general, network 810 canbe any combination of connections and protocols that supportcommunications between two or more computers and related devices.

Peripheral interface 855 may allow for input and output of data withother devices that may be connected locally with computer 805. Forexample, peripheral interface 855 may provide a connection to externaldevices 860. External devices 860 may include input devices, e.g., anyor all of the devices in the information acquisition means 10 and/orother suitable input devices, and output devices, e.g., any or all ofthe various actuator devices AC and/or other suitable output devices,e.g., a speaker. External devices 860 may also include portablecomputer-readable storage media such as, for example, thumb drives,portable optical or magnetic disks, and memory cards. Software and dataused to practice embodiments of the present disclosure, for example,program 848, may be stored on such portable computer-readable storagemedia. In such embodiments, software may be loaded onto non-volatilestorage 845 or, alternatively, directly into memory 840 via peripheralinterface 855. Peripheral interface 855 may use an industry standardconnection, such as RS-232 or Universal Serial Bus (USB), to connectwith external devices 860.

Display interface 865 may connect computer 805 to display 870, e.g., ahead-up display or a screen of a car navigation system. Display 870 maybe used, in some embodiments, to present a command line or graphicaluser interface to a user of computer 805. Display interface 865 mayconnect to display 870 using one or more proprietary or industrystandard connections, such as VGA, DVI, DisplayPort and HDMI.

As described above, network interface 850, provides for communicationswith other computing and storage systems or devices external to computer805. Software programs and data discussed herein may be downloaded from,for example, remote computer 815, web server 820, cloud storage server825 and compute server 830 to non-volatile storage 845 through networkinterface 850 and network 810. Furthermore, the systems and methodsdescribed in this disclosure may be executed by one or more computersconnected to computer 805 through network interface 850 and network 810.For example, in some embodiments the systems and methods described inthis disclosure may be executed by remote computer 815, computer server830, or a combination of the interconnected computers on network 810.

Data, datasets and/or databases employed in embodiments of the systemsand methods described in this disclosure may be stored and or downloadedfrom remote computer 815, web server 820, cloud storage server 825 andcompute server 830.

The embodiment is described above by way of example only and is notintended to limit the scope of this disclosure. The scope of thisdisclosure is defined by the claims, and modifications and alterationsbelonging to the scope equivalent to the scope of the claims all fallwithin the scope of this disclosure.

1. A driver condition estimating device configured to estimate conditionof a driver who is driving a vehicle, the driver condition estimatingdevice comprising: circuitry configured to: measure movement of a headof the driver from output of a camera that photographs the driver; anddetect a sign of abnormality of the driver from the movement of thehead; calculate a periodic feature amount from time series data showingthe movement of the head of the driver; calculate coherence between themovement of the head of the driver and lateral acceleration acting onthe head of the driver; calculate time series variation patterns fromthe calculated periodic feature amount and the calculated coherence; andcompare the calculated time series variation patterns with apredetermined threshold to determine existence of the sign ofabnormality of the driver.
 2. The driver condition estimating deviceaccording to claim 1, wherein the detector is configured to obtainlateral acceleration acting on the head of the driver, from output of asensor for measuring a movement state of the vehicle.
 3. The drivercondition estimating device according to claim 2, wherein the circuitryis configured to: recognize a head of the driver in a photographic imageobtained by the camera, and measure movement of the head of the driverfrom the photographic image, wherein the movement of the head of thedriver includes angles of the head.
 4. The driver condition estimatingdevice according to claim 2, wherein the circuitry is configured to:calculate periodic feature amounts from the time series data of movementof the head, and calculate autocorrelation indexes as the periodicfeature amounts by detrended fluctuation analysis.
 5. The drivercondition estimating device according to claim 2, wherein the circuitryis configured to: output an inquiry to the driver in response to theabnormality being determined, wherein the inquiry is at least one of avocal inquiry output by a speaker in the vehicle and a visual inquiryshown on a display in the vehicle.
 6. The driver condition estimatingdevice according to claim 2, wherein the detector is configured to:dimensionally reduce time series data of the calculated periodic featureamount and of the calculated coherence by using a nonlineardimensionality reduction method to obtain two-dimensional data as thetime series variation pattern; and determine existence of the sign ofabnormality of the driver from the obtained two-dimensional data byusing a determination line in a two-dimensional map as the predeterminedthreshold.
 7. The driver condition estimating device according to claim6, wherein the circuitry is configured to: recognize a head of thedriver in a photographic image obtained by the camera, and measuremovement of the head of the driver from the photographic image, whereinthe movement of the head of the driver includes angles of the head. 8.The driver condition estimating device according to claim 6, wherein thecircuitry is configured to: calculate periodic feature amounts from thetime series data of movement of the head, and calculate autocorrelationindexes as the periodic feature amounts by detrended fluctuationanalysis.
 9. The driver condition estimating device according to claim6, wherein the circuitry is configured to: output an inquiry to thedriver in response to the abnormality being determined, wherein theinquiry is at least one of a vocal inquiry output by a speaker in thevehicle and a visual inquiry shown on a display in the vehicle.
 10. Thedriver condition estimating device according to claim 1, wherein thedetector is configured to: dimensionally reduce time series data of thecalculated periodic feature amount and of the calculated coherence byusing a nonlinear dimensionality reduction method to obtaintwo-dimensional data as the time series variation pattern; and determineexistence of the sign of abnormality of the driver from the obtainedtwo-dimensional data by using a determination line in a two-dimensionalmap as the predetermined threshold.
 11. The driver condition estimatingdevice according to claim 1, wherein the circuitry is configured to:recognize a head of the driver in a photographic image obtained by thecamera, and measure movement of the head of the driver from thephotographic image, wherein the movement of the head of the driverincludes angles of the head.
 12. The driver condition estimating deviceaccording to claim 11, wherein the circuitry is configured to: calculateperiodic feature amounts from the time series data of movement of thehead, and calculate autocorrelation indexes as the periodic featureamounts by detrended fluctuation analysis.
 13. The driver conditionestimating device according to claim 12, wherein the circuitry isconfigured to: output an inquiry to the driver in response to theabnormality being determined, wherein the inquiry is at least one of avocal inquiry output by a speaker in the vehicle and a visual inquiryshown on a display in the vehicle.
 14. The driver condition estimatingdevice according to claim 11, wherein the circuitry is configured to:output an inquiry to the driver in response to the abnormality beingdetermined, wherein the inquiry is at least one of a vocal inquiryoutput by a speaker in the vehicle and a visual inquiry shown on adisplay in the vehicle.
 15. The driver condition estimating deviceaccording to claim 1, wherein the circuitry is configured to: calculateperiodic feature amounts from the time series data of movement of thehead, and calculate autocorrelation indexes as the periodic featureamounts by detrended fluctuation analysis.
 16. The driver conditionestimating device according to claim 1, wherein the circuitry isconfigured to: output an inquiry to the driver in response to theabnormality being determined, wherein the inquiry is at least one of avocal inquiry output by a speaker in the vehicle and a visual inquiryshown on a display in the vehicle.
 17. A driver condition estimatingmethod for estimating condition of a driver who is driving a vehicle,the driver condition estimating method comprising: calculating aperiodic feature amount from time series data showing movement of a headof the driver; calculating coherence between the movement of the head ofthe driver and lateral acceleration acting on the head of the driver;calculating time series variation patterns from the calculated periodicfeature amount and the calculated coherence; and comparing thecalculated time series variation patterns with a predetermined thresholdto determine existence of a sign of abnormality of the driver.
 18. Anon-transitory computer readable storage including computer readableinstructions that when executed by a controller cause the controller toexecute a driver state determination method for a driver in a vehicle,the method comprising: calculating a periodic feature amount from timeseries data showing movement of a head of the driver; calculatingcoherence between the movement of the head of the driver and lateralacceleration acting on the head of the driver; calculating time seriesvariation patterns from the calculated periodic feature amount and thecalculated coherence; and comparing the calculated time series variationpatterns with a predetermined threshold to determine existence of a signof abnormality of the driver.